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Abstract. Convolutional neural networks have significantly improved the field 
of computer vision. Most of the standard deep learning models for vision are 
based on convolutional neural networks. One such model is DenseNet, which is 
well known for providing comparable results to ResNet with fewer trainable var- 
iables. This paper proposes a mechanism to improve the performance of Dense- 
Net architecture. We have compared our proposed approach with the original 
DenseNet architecture on CIFAR100 dataset. Effects of applying dropout mech- 
anism with our proposed mechanism is also studied on DenseNet model and the 
obtained results showed that our approach helps DenseNet to significantly im- 
prove the performance and helping the model to learn faster. Even in situations 
where original DenseNet-121 would have overfit, our mechanism helped Dense- 
Net-121 to keep improving the accuracy without overfitting on the dataset assur- 
ing better accuracy on testing data. 
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tion. 


1 Introduction 


Convolutional neural networks have become base architecture for many of the top per- 
forming deep learning models in vision [1,2,3] because of the ability to work without 
feature engineered [4] input data. This ability has made many vision tasks [5,6,7] pos- 
sible which earlier were difficult to achieve using traditional approaches. DenseNet [8] 
is one such model based on convolutional neural networks and is known for providing 
comparable results to other top performing models with fewer trainable variables. Re- 
sidual networks helped many convolutional neural network models to improve their 
performance by enabling faster gradient flow during back propagation [9,10]. Dense- 
Net performs comparatively better than model based on traditional residual networks 
due to the presence of densely connected network in dense blocks because these dense 
connections enable each convolutional layer to be connected to every other convolu- 
tional layer present before it in the same block which form dense connectivity in the 
model as represented in Fig. 1. 


Fig. 1. Connectivity present in a dense block of DenseNet with 7 convolutional layers. 


In this paper, we have presented an approach to improve the performance of DenseNet 
architecture by adding some strategic connections to already densely connected convo- 
lutional layers present in the dense block. 


2 Binary Search Connections 


Layers present in dense block of DenseNet architecture are connected in such a manner 
that each layer is connected to every other layer present before it and feature maps of 
all preceding layers serve as an input to the current layer while the feature maps pro- 
duced by current layer serve as an input to the subsequent layers. This arrangement 
creates dense connectivity among layers present in the blocks and during back propa- 
gating the network learn to identify relevant connections to make a correct decision by 
adjusting value of weight and biases of these layers. In this paper we have assumed this 
a searching problem and proposed to add some strategic connections to specific layers 
of densely connected blocks to make the network more search able by increasing the 
number of connections to specific layers. 


Binary search is an algorithm used for searching an element in an already sorted list 
by taking minimum number of steps in sequential manner [11]. In this paper, we have 
proposed binary search connections which are based on the ideology of binary search 
algorithm. These connections provide better search ability in the network by adding 
connections to specific layers in the densely connected block of DenseNet. 


2.1. Implementation of binary search connections 


Binary search connections provide better search ability to the network by adding more 
number of connections to specific layers in dense block. These additional connections 
are designed using our simple but effective approach. 


Table 1. Parameter related details. 


S.no Parameter Initial Value 

1 Start 0. 

2 End Number of layers in dense block — 1. 

3 feteegs List containing empty lists equal to number of 


layers present in a dense block. 


Binary_Search_Connection(start, end, list_keys) 


if end-start>0 
then 
mid=(startt+end)/2 
if start!=end 
then 
if mid-start>2 
then 
list_keys[mid].append(start+1) // send output of “start+1” to “mid” as input 
end if 
end if 
if end-mid>2 
then 
list_keys[end].append(mid+1) // send output of “mid+1” to “end” as input 
end if 
Binary_Search_Connection(start+1, mid-1, list_keys) 
Binary_Search_Connection(mid+1, end-1, list_keys) 
end if 


At the time of designing dense connections inside a block keep track of the index of 
current layer and connect that current layer to all those previous layers whose index are 
mentioned at current index position in list_keys. An updated list_keys is calculated for 
each dense block by providing an empty list_keys containing empty lists equal to total 
number of layers present in that block to Binary_Search_Connection function which is 
later used for adding strategic connections to the layer as represented in Fig. 2. 


Fig. 2. DenseNet block with Binary Search Connection (represented in red color). 


3 Comparative Performance Analysis 


To check the effectiveness of our proposed mechanism we compared it with DenseNet 
architecture and have also studied the effect of applying dropout [12] on model perfor- 
mance while maintaining a fair comparison throughout the process by keeping few 
model parameters constant. Source code and result of all the experiments are available 
at our github repository [13]. 


3.1 Comparative study without using dropout 


Study is performed using CIFAR100 dataset [14] and the obtained results suggested 
that our approach helped DenseNet in significantly improving performance during 
training phase as shown in Fig. 3 and 4, validation phase as shown in Fig. 5 and 6, and 
inference phase by improving search ability in the dense block. 


Table 2. Experiment related details. 


S.no Parameter Value Details 

1 Dataset CIFAR100 Dataset 

2 Number of images in Training Data 50,000 Number of training images 
3 Number of images in Testing Data 10,000 Number of testing images 
4 Weight decay rate 0.0001 Constant value 

5 Number of Dense Blocks 4 Constant value 

6 Growth rate 32 Constant value 

7 Number of layers in each dense block 12 Constant value 

8 Dropout value 0% No dropout 
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Fig. 3. Comparative analysis of training loss on CIFAR 100 dataset. 
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Fig. 4. Comparative analysis of training accuracy on CIFAR 100 dataset. 
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Fig. 5. Comparative analysis of validation loss on CIFAR 100 dataset. 
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Fig. 6. Comparative analysis of validation accuracy on CIFAR 100 dataset. 


Table 3. Performance on testing data. 


Parameter DenseNet DenseNet with Binary search connections 
(BSC-DenseNet) 

Loss 2.955 2.926 

Accuracy 32.3 % 35.9 % 


3.2 Comparative study using dropout 


Dropout [12] is a mechanism used as regularization to avoid over-fitting in a network. 
The effect of applying dropout is studied on larger DenseNet-121 model in training 
phase as shown in Fig. 7 and 8, validation phase as shown in Fig. 9 and 10, and infer- 
ence phases. After doing proper analysis it was found that even with applying dropout 
mechanism on larger architecture of DenseNet-121 our approach significantly helped 
DenseNet-121 to improve its performance. 


Table 4. Experiment related details. 


S.no Parameter Value Details 

1 Dataset CIFAR100 Dataset 

2 Number of images in Training Data 50,000 Number of training images 
3 Number of images in Testing Data 10,000 Number of testing images 
4 Weight decay rate 0.0001 Constant value 

5 Number of Dense Blocks 4 Constant value 

6 Growth rate 24 Constant value 

7 Number of layers in dense block 1 6 Constant value 

8 Number of layers in dense block 2 12 Constant value 

9 Number of layers in dense block 3 24 Constant value 

10 Number of layers in dense block 4 16 Constant value 

11 Dropout value 20 % 20 percent dropout 
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Fig. 7. Comparative analysis of training loss on DenseNet-121 architecture during training 
phase with dropout of 20 percent. 
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Fig. 8. Comparative analysis of training accuracy on DenseNet-121 architecture during training 
phase with dropout of 20 percent. 
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Fig. 9. Comparative analysis of validation loss on DenseNet-121 architecture during training 
phase with dropout of 20 percent. 
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Fig. 10. Comparative analysis of validation accuracy on DenseNet-121 architecture during train- 
ing phase with dropout of 20 percent. 
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After 15 epochs, original DenseNet-121 and our BSC-DenseNet-121 (DenseNet-121 
with binary search connection) performances are compared using proper graphical anal- 
ysis on the result and after analyzing it was found that in DenseNet-121 overfitting 
occurred after 12 epochs while the BSC-DenseNet-121 keeps on improving training 
and validation accuracies without overfitting or underfitting on the training data. This 
characteristic helped BSC-DenseNet-121 to obtain better performance on testing data 
than the original DenseNet-121 model. 


Table 5. Performance on testing data. 


Parameters DenseNet-121 DenseNet-121 with Binary search connections 
(BSC-DenseNet) 

Loss 3.308 3.290 

Accuracy 23.2 % 30.6 % 


4 Conclusion 


Our proposed approach helps densely connected networks to significantly improve per- 
formance on test data by reducing training and validation loss and increasing accuracy 
during training and validation phases which in turn improved model performance at 
inference time. Binary search connection based DenseNet handle overfitting better than 
original DenseNet model. When the dropout mechanism is applied to DenseNet with 
binary search connection it showed better performance than the original DenseNet 
model with dropout mechanism. Hence, binary search connections can be used in 
DenseNet to improve model performance in training, validating, and testing phases. 
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